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-- The MAILING DATE of this communication appears on the cover sheet with the correspondence address •• 
Period for Reply 



A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH(S) FROM 
THE MAILING DATE OF THIS COMMUNICATION. 

- Extensions of time may be available under the provisions of 37 CFR 1.136(a). In no event, however, may a reply be timely filed 
after SIX (6) MONTHS from the mailing date of this communication. 

- If the period for reply specified above is less than thirty (30) days, a reply within the statutory minimum of thirty (30) days will be considered timely. 

- If NO period for reply is specified above, the maximum statutory period will apply and will expire SIX (6) MONTHS from the mailing date of this communication. 

- Failure to reply within the set or extended period for reply will, by statute, cause the application to become ABANDONED (35 U.S.C. § 133). 
Any reply received by the Office later than three months after the mailing date of this communication, even if timely filed, may reduce any 
earned patent term adjustment. See 37 CFR 1.704(b). 

Status 

I )I3 Responsive to communication(s) filed on 21 September 2001 . 
2a)D This action is FINAL. 2b)S This action is non-final. 

3) D Since this application is in condition for allowance except for formal matters, prosecution as to the merits is 

closed in accordance with the practice under Ex parte Quayle, 1935 CD. 1 1 , 453 O.G. 213. 

Disposition of Claims 

4) ^ Claim(s) 1-32 is/are pending in the application. 

4a) Of the above claim(s) is/are withdrawn from consideration. 

5) D Claim(s) is/are allowed. 

Q)M Claim(s) 1.3-11.14-19 and 22-32 is/are rejected. 
7)I3 Claim(s) 2.12.13.20 and 21 is/are objected to. 
- 8)E]- Claim(s)"__ are r subject "to" restriction and/or election requirement. 

Application Papers 

9)D The specification is objected to by the Examiner. 

10)D The drawing(s) filed on is/are: a)D accepted or b)D objected to by the Examiner. 

Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1.85(a). 
Replacement drawing sheet(s) including the correction is required if the drawing(s) is objected to. See 37 CFR 1.121(d). 

I I )□ The oath or declaration is objected to by the Examiner. Note the attached Office Action or form PTO-152. 

Priority under 35 U.S.C. § 119 

12)D Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 119(a)-(d) or (f). 
a)D All b)D Some * c)D None of: 

1 .□ Certified copies of the priority documents have been received. 

2. D Certified copies of the priority documents have been received in Application No. . 

3. D Copies of the certified copies of the priority documents have been received in this National Stage 

application from the International Bureau (PCT Rule 17.2(a)). 
* See the attached detailed Office action for a list of the certified copies not received. 
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DETAILED ACTION 

This communication is in response to the document dated 4/8/03. Claims 1-32 are 
pending in this application. Claims 1, 7, 11, 19, 27 and 32 are independent claims. 

Claim Rejections - 35 USC §112 

The following is a quotation of the second paragraph of 35 U.S.C. 112: 

The specification shall conclude with one or more claims particularly pointing out and distinctly 

claiming the subject matter which the applicant regards as his invention. 

Claim 14 recites the limitation "said" in the limitation "said weight". There is 

insufficient antecedent basis for this limitation in the claim. 



Claim Objections 

The claims 1, 7, are objected to because they include reference characters, 
which are not enclosed within parentheses. 

Reference characters corresponding to elements recited in the detailed 
description of the drawings and used in conjunction with the recitation of the same 
element or group of elements in the claims should be enclosed within parentheses so 
as to avoid confusion with other numbers or characters which may appear in the claims. 
See MPEP§ 608.01 (m). 
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Claim Rejections - 35 USC § 103 

The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

Claims 1,4-6, 14-15, 19 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Liu et al. (US 6675159) and further in view of Spencer (US 
5826261). _ - 

As per claim 1 , Lin et al. teach searching multimedia documents from an ontology - col. 
6, lines 32-51 ; col. 8, lines 43-65; ontological data and databases - col. 1 1 , lines 5-44. 

In the specification, page 2, Applicants state that "ontology: the hierarchical 
structuring of knowledge about objects by sub-categorizing based on their relevant 
qualities "; "" annotation" and "keyword" eguivalent ". On page 9 of the specification, last 
paragraph, Applicants state, that " llndex treats these leaning instances as a bag of 
words to be indexed ..." 

Lin et al. teach clustering, ontology and nodes - col. 13, lines 25-50. 

learning data preparation component accessing mappings of annotations in said 

ontology and fusing annotations mapped in each of said nodes to form learning 
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instances - col. 10, lines 13-41 ; col. 13, lines 25-39; col. 16, lines 27-40; col. 18, lines 
17-30; col. 24, lines 41-59. 

searching, queries and data retrieval using inverted index, and ranking of index 
documents- col. 1, lines 34-67; col. 11, lines 25-44; term frequency- col. 3, lines 35- 
44; ontological concepts and adaptive weights allocated to each node - col. 18, lines 
17-32; classifying new documents, form new training set, new concept is added to the 
ontology -col. 10, lines 48-61; col. 11, lines 25-44; col. 12, lines 11-29. 
an information retriever extracting information related to said requested 
annotations/keywords from said most relevant nodes and said one or more databases 
over said network - col. 12, line 30 to col. 13, line 25. 

a contextual information linker linking multimedia content with said extracted information 
- col. 1, lines 34-67; col. 21, lines 24-29. 

However, Lin et al. do not explicitly teach inverse document frequency and 
contribution frequency. Spencer teaches text or information retrieval from multiple, 
distributed database - col. 1, lines 10-11; local relative contribution that various term 
would have on the results of a query within each of the document databases; term 
frequency, and idf - col. 3, lines 25-40; inverted index and a contribution table - col. 10, 
lines 11-19; col. 11, line 25 to col. 12, line 19. Thus, it would have been obvious to one 
of ordinary skill in the art at the time of the invention to combine Liu et al.'s teaching and 
Spencer et al.'s teaching to better analyze and map multimedia annotations/keywords to 
ontologies. 
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As per claim 4, Liu et al. teach access to the ontology data - col .11, lines 25-43; end 
user interface -fig. 1, item 114. However, Liu et al. do not explicitly teach wherein said 
annotations are accessible via any of the following devices: an interactive television, a 
computer... , a telephone. Spencer teaches the client computer 101 direct the user's 
query to the query router - col . 1 8, lines 30-61 . Thus, it would have been obvious to 
one of ordinary skill in the art at the time of the invention to combine Liu et al.'s teaching 
and Spencer et al.'s teaching in order to allow efficient accesses to ontologies from 
users 1 devices. 

As per claim 5, Liu et al. teach wherein said network is any of the following: WAN, LAN, 
wireless network, Internet, or telephony network -col. 1, lines 15-22; col. 20, -lines 10- 
19. 

As per claim 6, Liu et al. teach tokenizing said learning instances - col. 16, lines 27-40; 
stemming said tokenized learning instances - col. 12, lines 65-66; stop-word filter - col. 
14, lines 33-49. 

As per claim 14, Lin et al. do not explicitly teach contribution frequency. Spencer 
teaches text or information retrieval from multiple, distributed database - col. 1 , lines 
10-11; local relative contribution that various term would have on the results of a query 
within each of the document databases; term frequency, and idf - col. 3, lines 25-40; 
inverted index and a contribution table -col. 10, lines 11-19; col. 11, line 25 to col. 12, 
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line 19; error contribution weight - col. 5, lines 13-33; frequency of terms and weight 
pair -col. 6, lines 43-63; col. 13, lines 30-64; col. 14, lines 52-62. Thus, it would have 
been obvious to one of ordinary skill in the art at the time of the invention to combine Liu 
et al.'s teaching and Spencer et al.'s teaching to better analyze and map multimedia 
annotations/keywords to ontologies. 

As per claim 1 5, Lin et al. teach wherein said annotations are retrieved from any of the 
following sources: text documents, message boards, chat rooms, product descriptions, 
and multimedia documents...- col. 10, lines 17-33; col. 14, lines 23-44. 

As per claim 19, Lin et al. teach a server receiving requests Jor contextual information - 
for a selected multimedia representation - col. 9, lines 28-47; searching multimedia 
documents from an ontology - col. 6, lines 32-51 ; col. 8, lines 43-65; ontological data 
and databases — col .11, lines 5-44; mappings of annotations in said ontology and 
annotations mapped in each of said nodes - col. 10, lines 13-41; col. 13, lines 25-39; 
col. 16, lines 27-40; col. 18, lines 17-30; col. 24, lines 41-59; searching, queries and 
data retrieval using inverted index, and ranking of index documents - col. 1 , lines 34-67; 
col. 1 1 , lines 25-44; ontological concepts and adaptive weights allocated to each node - 
col. 18, lines 17-32; classifying new documents, forming new training set, and new 
concept is added to the ontology -col. 10, lines 48-61; col. 11, lines 25-44; col. 12, lines 
11-29; renders said retrieved information comprising said multimedia and said retrieved 
contextual information - col. 7, lines 10-23; col. 11, lines 25-44. 
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Spencer further improve Lin et al.'s teaching of a server receiving requests for 
contextual information... at col. 8, lines 42-52; col. 9, line 15 to col. 10, line 19. Spencer 
teaches text or information retrieval from multiple, distributed database - col. 1, lines 
10-11; local relative contribution that various term would have on the results of a query 
within each of the document databases- col. 3, lines 25-40. Thus, it would have been 
obvious to one of ordinary skill in the art at the time of the invention to combine Liu et 
al.'s teaching and Spencer et al.'s teaching in order to allow users to better serve users 
in data retrieval. 

Claim 3 is rejected under 35 U.S.C. 103(a) as being unpatentable over Liu et al. 
(US 6675159), Spencer (US 5826261), and further in view of Liddy et al. (US 
6304864). 

As per claim 3, Liu et al. teach: "inverted index that links keywords with the documents 
in which they appear. Documents with the most keywords that match the input query 
are retrieved." - col. 1, line 34-48; col. 13, lines 25-40; col. 26, lines 42-48; text content 
- col. 7, lines 11-22; col. 10, lines 17-33. However, Liu and Liddy et al. do not explicitly 
disclose wherein said multimedia documents comprises audio, graphics, video 
documents. Liddy et al. teach a system for retrieving multimedia information from a 
computer-based network, such as Internet -col. 1, lines 11-22; "the information of the 
documents retrieved may be of one or more multiple media types, such as text, graphic, 
audio, video, or any, as defined in the user profile" - col. 4, lines 52-55; key terms and 
frequency of the subject categories - col. 1 1 , lines 9-26; automatic relevant 
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determination - col. 12, lines 32-41 . Thus, it would have been obvious to one of 
ordinary skill in the art at the time of the invention to combine Liu et al.'s teaching, 
Spencer et al.'s teaching, and Liddy et al.'s teaching to efficiently map all digital 
document types 1 annotations/keywords to ontologies. 

Claim Rejections - 35 USC § 102 

The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that 

form the basis for the rejections under this section made in this Office action: 
A person shall be entitled to a patent unless - 

(e) the invention was described in (1) an application for patent, published under section 122(b), by 
another filed in the United States before the invention by the applicant for patent or (2) a patent 
granted on an application for patent by another filed in the United States before the invention by the 
applicant for patent, except that an international application filed under the treaty defined.in section 
351(a) shall have the effects for purposes of this subsection of an application filed in the United States 
only if the international application designated the United States and was published under Article 21(2) 
of such treaty in the English language. 

Claim 27 is rejected under 35 U.S.C. 102(e) as being aniticipated by Lin et al. 
(US 6675159). 

As per claim 27, Lin et al. teach extracting annotations from a multimedia document 
segment -col. 12, lines 11-18; col. 16, lines 51-67. 

mapping said extracted multimedia document segment to an appropriate node in said 
ontology - col. 13, lines 25-40; col. 18, lines 9-51. 

comparing to other related content mapped to said appropriate node - col. 1, lines 34- 
67; col. 9, lines 2-8; col. 25, lines 58-67; col. 26, lines 42-48. 
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clustering, ontology and nodes - col. 13, lines 25-50; classifying new documents, form 
new training set, new concept is added to the ontology - col. 10, lines 48-61 ; col. 1 1 , 
lines 25-44; col. 12, lines 11-29. 

linking multimedia content with said extracted information - col. 1 , lines 34-67; col. 21 , 
lines 24-29. 

Claim 28 is rejected under 35 U.S.C. 103(a) as being unpatentable over Liu et al. (US 
6675159), and further in view of Liddy et al. (US 6304864). 

As per claim 28, Lin et al. teach classifying new documents, forming new training set, 
and new concept is added to the ontology - col. 10, lines 48-61 ; col. 1 1, lines 25-44; 
col. 12, lines 11-29. (Examiner did not find the definition of the limitation: pre- 
certification in the specification.) However, Liddy et al. further teach the agent leader 
...continuously updates the rank as new documents are retrieved by agents - col. 1 1 , 
lines 27-39; automatically update ranking of relevant documents based on users' 
selections - col. 3, lines 40 to col. 4, line 28. Thus, it would have been obvious to one of 
ordinary skill in the art at the time of the invention to combine Liu et al.'s teaching, 
Spencer's teaching with Liddy et al.'s teaching to better update documents ranks based 
on relevancy, user profiles, etc... to better provide importance or most relevant 
documents to users. 

Claims 7-11, 16-18, 24-26, 29-32 claims the same subject matters as of claims 1, 3-6 
and are rejected based on the same ground of rejection as of claims 1 , 3-6. 
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Allowable Subject Matter 

Claims 2, 12-13, 20-21 are objected to as being dependent upon a rejected base 
claim, but would be allowable if rewritten in independent form including all of the 
limitations of the base claim and any intervening claims. 

Conclusion 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to LINH BLACK whose telephone number is 571-272- 
4106. The examiner can normally be reached on.8am - 5pm.- - — 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, JOHN BREENE can be reached on 571-272-4107. The fax phone number 
for the organization where this application or proceeding is assigned is 703-872-9306. 

Information regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
Status information for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should 
you have questions on access to the Private PAIR system, contact the Electronic 
Business Center (EBC) at 866-217-9197 (toll-free). 

LINH BLACK 
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